A network data stream is composed of a plurality of frames. A frame is a logical unit of data organized specifically for transmission. In the prior art, a frame may also be referred to as a packet, block, or cell. In a network data stream, a complete frame is composed of a header followed by a payload that is followed by a trailer. A header contains a flag bit, or set of bits, to indicate the beginning of the frame followed by control data and address data such as synchronizing bits, address of the destination or target device, address of the originating device, length of frame, and so on. A payload comprises the data to be transmitted and, depending on the transmission protocol, may be of fixed or variable length. A trailer contains error detection and correction bits and a set of bits to indicate the end of the frame. Frames are assembled by the sending computer and placed in a network data stream to be transmitted to a receiving computer via a network. Frames are extracted from network data stream by the receiving computer. The receiving computer extracts and uses the payload of the frame.
The assembly, transmission, and extraction of frames and extraction of payloads from frames are governed by standard sets of rules called protocols. A network protocol, i.e., protocol, is a set of rules used by computers to communicate via a network. Protocols enable computers attempting to exchange data to “understand” one another. Protocols are sometimes described as “languages;” however, a protocol is more like the syntax of a language, i.e., the order in which words are put together, than the language itself. In order for two computers to communicate, each computer must understand the protocols used by the other computer.
To ensure that computers understand protocols used by other computers, standard protocols have been developed. Organizations such as the International Standards Organization (ISO) are charged with the definition, control, and publishing of the specifications of standard protocols. This makes protocols available to vendors that want to create products that adhere to standard protocols. To aid vendors in implementing standard protocols, architectural models are provided. A model provides an easy-to-understand description of the networking architecture and serves as the framework for the standards. The Open System Interface (OSI) model is an example of such a model.
The OSI model is layered. In the OSI model, protocols are organized into a stack of layers called a protocol stack (i.e. a stack). The layers are used to encapsulate and organize the functions required to generate and manage frames. Each of the layers uses the services of the layer below to build an “enriched service,” i.e., a more capable service. The layered approach provides a logical division of responsibility. Each layer handles prescribed functions. Such layering can be compared to an automobile assembly line. At points along the assembly line, a handle is fitted to a door, a door is fitted to a chassis, and so on. This assembly line approach applied to protocol layers allows each protocol layer to specialize in the function provided by the layer. Such function specialization makes it easier to implement protocols and if a communication problem occurs, the problem can be isolated to a specific layer.
Using a stack of protocol layers to send a payload in a frame is analogous to sending a letter in an envelope to a friend in another city. As illustrated in FIG. 1A, the layers of transport are analogous to the protocol layers. The desired communication is between you 100 and your friend 115. Because you 100 and your friend 115 are separated by distance, you 100 cannot hand the letter directly to your friend 115. Instead, you 100 place the letter in an envelope and give the envelope to your post office 105. Your post office 105 gives the envelope to an airline 110. The airline 110 places the envelope in a shipping container and transports the shipping container to your friend's post office 120. Your friend's post office 120 removes the envelope from the shipping container and delivers the envelope to your friend 115. Your friend 115 opens the envelope and reads your letter. You do not need to find your friend's house in the distant city. That is the responsibility of the post office in that city. You only need to specify your friend's address. The post office is not concerned with how to fly an airplane. That is the responsibility of the airline. Each layer assumes that the layer below it will provide certain functions. Each layer provides additional functionality. Similarly, at each layer of a protocol stack information is added to a frame passed from layer to layer that relates to the function of the layer.
FIG. 1B shows the stacks of two exemplary computing devices Computer A 200 and Computer B 240. It can be seen that the stacks of both computing devices have the same layers: application 205 and 245; presentation 210 and 250; session 215 and 255; transport 220 and 260; network 225 and 265; and data link 230 and 270. The two stacks are connected by a physical (network hardware) layer 235. A frame that is to be sent from Computer A 200 to Computer B 240 is assembled by placing information into a frame without protocol headers, i.e., a payload, passing the frame down the stack on Computer A 200, adding headers to the frame at each layer, and sending the frame on the physical hardware layer 235. More specifically, as the frame is passed down the Computer A stack, information concerning the protocol of each layer is added to the frame. Included in the information added at each layer is information indicating which protocol is used in the layer immediately above the layer. The protocol information is used later as the frame is passed up the protocol stack of Computer B 240 and disassembled. Each layer of the stack on Computer B 240 reverses the process of the associated layer of the stack of Computer A thereby extracting the frame from physical hardware layer 235, i.e., the network, and extracting the payload from the frame.
The stacks of the two exemplary computing devices 200 and 240 in FIG. 1A are examples of OSI protocol stacks. Using the stack in Computer A 200 as an example of an OSI protocol stack, the application layer 205 provides network services such as X.400 email, HyperText Transport Protocol (HTTP), File Transfer Protocol (FTP), and telnet. The presentation layer 210 converts the information in the frame into data formatted using the format recognized by Computer A 200. The session layer 215 establishes a session, i.e., a series of information exchanges, between Computer A 200 and Computer B 240. The transport layer 220 multiplexes data streams from different applications. Those skilled in the art will appreciate that multiplexing is a technique to transmit a number of separate signals simultaneously over a single channel or line. The transport layer 220 may also provide error correction. An example of a transport protocol usable in the transport layer 220 is the Transmission Control Protocol (TCP). The network layer 225 finds a route for frames to take through the network and directs frames to the correct computer. An example of a network protocol usable in the network layer 225 is the Internet Protocol (IP). The data link layer 230 is the logical, as opposed to physical, data link. The data link layer 230 provides media access control, detects and corrects errors on the physical link, and provides control of the flow of data. For local area networks (LANs) where all computers share a communications media, this layer determines which node is allowed to transmit. Examples of data link layer protocols are Ethernet and Token Ring. The physical layer 235 defines the characteristics of the physical connections, such as type of wire, plug shape, how a zero bit and one bit are represented, what voltages are used, and so on. The physical is the only layer that actually sends bits to another computer. Examples of physical layer protocols usable in the physical layer 235 are SONET and RS-232C.
The network model used by the World Wide Web, i.e., the Internet model, probably the most commonly used network model, does not exactly follow the OSI model, but closely imitates the OSI model. Because the Internet model was designed to run on top of existing different networks, the Internet model does not define the lower layers of the OSI model. The Internet model layers are: Application, Transport, Internet, and Network Interface. The Application layer provides network services such as HTTP, FTP, and Telnet. The Transport layer multiplexes data streams from different applications and may also provide error correction. Examples of Transport layer protocols are TCP and User Datagram Protocol (UDP). The Internet layer provides routing services like Internet Protocol (IP). The Network Interface layer provides access to the Data Link and lower protocols like Ethernet.
In both OSI stacks and Internet stacks, each layer communicates with its peer layer by prefixing the data from the above layer with a header as shown in FIG. 1C. FIG. 1C shows the header prefixes for the OSI stack model. AH represents the application layer 205 header. PH represents the presentation layer 210 header. SH represents the session layer 215 header. TH represents the transport layer 220 header. NH represents the network layer 225 header. DH represents the data link layer 230 header. The physical layer is represented by the lower layer of FIG. 1C with a PH outside the header boxes and a DT at the end of the data. More specifically, the data link layer often adds a trailer to the frame that contains a cyclic redundancy check (CRC) to detect errors. This addition is represented by DT in FIG. 1C. The physical layer may, or may not, append a header or trailer to the frame.
The bottom frame, i.e., complete frame, is sent across the physical network 235. When the frame is received at the other end, the headers are stripped off as the frame is passed up the stack to the user application. Each layer provides functions or services for the layer above it. Each layer calls upon services provided by the layer below it. The layers are implemented in each computer on the network. Each layer communicates with the layer's peer layer in another computer. Although the logical communication is between peer layers on different computers, the actual flow of information is down the protocol stack on the sending computer and up the protocol stack on the receiving computer. When a layer wants to send something to the layer's peer layer in another computer, the layer calls a function in the layer below it to actually send the data. Only the lowest layer actually sends bits to another computer.
An example is an email application on Computer A 200 sending an email message frame from an email message to Computer B 240. The email application on Computer A 200 operates in the topmost layer of the stack, i.e., application layer 205. The email application adds a header, e.g., an application header (AH), to the frame. The frame is passed to the next lower layer, i.e., the presentation layer 210, where the email application or possibly another software program or service adds another header, e.g., presentation header (PH), to the frame. This process continues through all the layers until the frame reaches the lowest layer, the physical hardware layer 235, where the frame is sent to the stack of Computer B 240. The data link layer 270 of the stack of Computer B 240 extracts the frame from the physical hardware layer 235. The frame passed up the stack of Computer B 240. At each layer the header relating to the layer is read, used to make decisions about what to do with the message, and removed. At the application layer 245 of the stack on Computer B 240, an email application receives the payload and uses the payload when reconstructing the original message.
If a network data stream, i.e., stream, is interrupted or if the information in the stream is corrupted, the stream must be analyzed to find the cause of the problem. The critical and often difficult parts of the task of analyzing a stream are breaking the stream into frames and organizing the information in the frames using protocol rules so humans can understand the information. A computer program used to capture frames is called a “network monitor” and is sometimes called a “network sniffer” by those skilled in the art. A computer software program used to interpret frames according to the rules of a protocol is called a “protocol parser” (i.e. parser). A computer program that uses one or more parsers to analyze a stream is called a “protocol analyzer.”
Network monitors capture, i.e., identify and extract, frames from a stream on a network and present the frames in a human-readable format. As will be readily appreciated by those skilled in the art and others the capture function is difficult to execute since a typical stream passes thousands of frames per second that a human network monitor user must analyze. To be useful, the captured frames must be narrowed down to only the frames related to a specific information exchange. A set of frames that comprise a specific information exchange is called a “conversation.” Functionally, a conversation is a set of frames that are related because each of the frames in the set of frames contains identifiers that are unique to the conversation. The identifiers are built from the headers added to the frame at each protocol layer as shown in FIG. 1C and described above. A conversation takes place in one protocol layer of the protocol stacks of the communicating computers. It is possible to assemble more than one conversation from the same set of frames because conversations may be assembled for each layer in a protocol stack. If the information that uniquely identifies the frames in a conversation can be identified, a filter can be constructed to capture only the frames in the conversation.
In the prior art, the network monitor user has been required to identify the information that uniquely identify the frames in a conversation. In the past, this has usually been done by capturing a small set of frames on a restricted network during a known information exchange and searching for common values in the frames. Because this approach is time consuming and thus, inherently expensive, any assistance a network monitor can provide to help identify conversations makes the network monitor a more useful tool.
A computer program, such as a network monitor, is usually developed by writing the computer instructions in a human-readable computer language and then compiling, i.e., translating, the computer instructions into a format computing devices are able to read. Such a format is referred to as a machine-readable format and a program that has been compiled into a machine-readable format is called machine code. A computer program compiled into machine code can be executed by a computing device. While parsers may be written as integral components of a network monitor prior to the network monitor being compiled into machine code, if the parser needs to be changed, the entire network monitor must be recompiled. To avoid excessive recompiling, a parser is usually written and compiled into an independent, reusable software module. An example of such a reusable software module is a dynamically linked library (DLL). A DLL is an independent, reusable software module with a well defined interface that allows a software program to attach and use, i.e., link, the DLL while the software program is executing. A network monitor links and uses parsers compiled into DLLs. A DLL can be recompiled without affecting the network monitor that links to the DLL.
In the prior art, parsers, i.e., parser DLLs, have been developed for each of the two hundred or so industry standard protocols. A network monitor has been developed to link to each parser. In addition to industry standard protocols, new protocols to address special needs are constantly being introduced requiring new parsers to be developed and linked into network monitors. Developing and maintaining parsers and writing the computer instructions in the network monitor to link to parsers is difficult, time consuming, and therefore, costly. A single parser usually comprises one thousand or more lines of computer instructions. After a parser is compiled into a DLL, only a description of the DLL's interface is available to the computer programmer who intends to link the DLL into a network monitor. Unless the computer programmer has access to the original parser computer instructions in the parser, it is impossible for the programmer to know the nature or quality of the DLL. Even with a copy of the original computer instructions, a computer programmer may need to study a parser for hours or even days to assess the robustness and security of a parser. The hidden nature of compiled DLLs causes other problems. Many parsers perform similar functions but, since the parsers, i.e., DLLs, are often developed by different computer programmers at different times, the computer instructions to implement the same or similar functions may have been rewritten dozens of times making the instructions difficult to interpret and often in error. Also, if a defect is discovered and corrected in one parser, the correction is often not propagated to other parsers performing similar functions.
The problems described above demonstrate a clear need for a way to more easily develop and maintain parsers for a large and growing plurality of protocols; more easily integrate parsers into a network monitor; reduce the number of computer instructions in each parser; make parsers more transparent to those who use parsers; centralize the common aspects of parsers; and assist in identifying conversations.